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SUMMARY 


This document presents the test report for the Onboard Navigation 
(ONAV) Console Expert/Trainer Entry Expert System. It includes a 
background summary of testing procedures used and summaries of 
test results. General conclusions are also presented. 
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Section 2 


INTRODUCTION 


2 . 1 PURPOSE 

This document describes the test results for the ONAV Entry phase 
system. A summary of the test methods and analysis results are 
included. 


2 . 2 APPLICABLE DOCUMENTS 

The following document represented the test plan which guided the 
activities described in this test report: 

Test Plan for the Onboard Navigation (ONAV) Console Expert/ 
Trainer System, Entry Phase, LinCom Corporation, for RICIS 
Proj . No. AI-8 , Coop. Agrmt. No. NCC9-16, Subcontract No. 
005, April 15, 1988. 

The test plan and this test report are obviously interrelated. 
Many sections of the test report correlate exactly to section in 
the test plan, where objectives and recommended procedures are 
given. The test plan should be examined concurrently with the 
test report to provide a comprehensive discussion of the ONAV 
Entry expert system testing. 

Additionally, this test report makes reference to aspects of the 
testing activity that are described in the following user's guide 
for the ONAV Entry system: 

User's Guide for the Onboard Navigation (ONAV) Console 
Expert/Trainer System, Entry Phase, Final Version, LinCom 
Corporation, April, 1988. 
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Section 3 


TEST RESULTS 


3.1 


BACKGROUND 
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3.2 RULE TESTS 


3.2.1 Inspection 
Analysis/Results : 

The XREF program was run for all the rule groups into which the 
Entry knowledge base had been broken into. Several typographical 
errors were found. These were subsequently corrected. It is 
important to note that, while we did not have a complete list of 
all known valid patterns, the context of the Entry phase expert 
system provided a basic guide as to what a correct and/or 
incorrect pattern was. 

A byproduct of this process was an increased awareness of the 
importance of looking at every part of a rule with the same 
rigorousness as the patterns themselves. Sensitivity to spelling 
of rule names for example, while not as significant functionally 
as rule patterns, was gained. Out of this came the realization 
that greater consistency was needed for the rule names within the 
various rule groups of the ONAV knowledge base. Such consistency 
aided cross referencing and more detailed correlations of rule 
information. Some renaming of rules was performed to accomplish 
this. 


3.2.2 Compilation 
Analysis/Results : 

The "(rules)" list generated by the CLIPS environment compared 
well with the XREF listing. No discrepancies were found. As was 
mentioned in section 3.2.1, the rule name inconsistencies were 
very evident when performing this test. 

In a sense, this test "exercised" both CLIPS and the XREF 
program. In doing the compilation to check that all rules 
compile, this certainly is useful from the standpoint of having 
readable CLIPS rules. On the other hand, having to check 
explicitly for duplicate rule names might not be necessary if 
this particular function was handled explicitly or implicitly by 
CLIPS or XREF. As it was during testing, we did not have complete 
information as to the extent of these capabilities in CLIPS and 
XREF. Therefore, this test to check rule names was necessary, and 
ensured that no problems of this type were present. 
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3.2.3 Default Data Activations 
Analysis/Results : 

During the initial preparation for this test, it was found that 
the ONAV rule file "control. r" had to be modified for this test. 
The modification was very simple. The rule that begins the rule 
control flow performs, among other things, the assertion of facts 
from the comps file into the rule base. For this test, this was 
not desired and was the opposite of what this test was trying to 
accomplish. Because of this situation, the "control. r" file used 
in this test had the call to "fact assertion" deleted (commented 
out actually) . In addition, the "single step" feature of ONAV 
(see user's guide reference in the applicable documents section 
2.2) was used to make the ONAV system execute one and only one 
complete cycle through the control flow. This usage can be seen 
in the listing of the test results. 

The test was run using the batch file shown in Appendix A. The 
results of running the batch file are provided in Appendix B. An 
examination of these results shows that the only rules that 
executed with the default data were control flow rules, with one 
exception noted in the IMU rules. The control flow rule action 
was to be expected and considered normal. The unexpected rule 
execution occurred in the IMU rule group. Here, IMU status light 
rules executed to control the expert system display. While 
initially unexpected, analysis showed this to be acceptable and 
appropriate given the values of the default data and the function 
of the rules that executed. 


3.2.4 Individual Rule Sets Loaded 
Analysis/Results : 

As testing activities progressed, this particular test as defined 
was determined to be of marginal or no value. Subsequently, a 
decision was made not to perform this test. The reasoning here 
was that: l) the original intent of this test and even more 
testing would be accomplished by a later series of tests; 
therefore, little or no benefit would be gained by doing this 
test, and 2) an excessive amount of preparation time is required 
to perform this test, in that each and every rule in the expert 
system would have to have a special data set developed to trigger 
the patterns in that rule. Taking this into consideration, and 
given that later testing would cover this, it was decided to 
eliminate this test. 
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3.2.5 


All Rule Sets Loaded 


Analysis/Results : 

This test was not performed, for the same reasons as described in 
section 3.2.4. The same rationale applied since the tests 
described in sections 3.2.4 and 3.2.5 are variations of each 
other. 


3.3 GROUP TESTS 


3.3.1 Functional Inspection 

Analysis/Results : 

This part of the testing involved detailed examination of rules 
and specification paperwork, as compared to most of the previous 
computer based testing performed. The rule base of the expert 
system was examined systematically, one group at a time. For each 
of these groups, the left and right hand side patterns and 
actions of each rule were compared to the Entry phase knowledge 
requirements specification: 

"Knowledge Requirements for the Onboard Navigation (ONAV) 

Console Expert/Trainer System," Entry Phase Specifications, 

Version 1.1, LinCom Corporation, April 1988. 

Numerous discrepancies in patterns were found, particularly in 
the form and syntax of patterns used in more than one group of 
rules. For example, a pattern used in the IMU rules was 
referenced in some of the TACAN rules, yet the pattern was not 
consistent in format in the two groups. In other situations, 
discrepancies between the code and the specification were 
ambiguous and required consultation with the personnel who 
provided the original inputs and expertise for the knowledge 
specifications. In several instances, the original thinking of 
the expert was not consistent with the expert's current thinking 
and therefore resulted in a modification to the specification as 
well as the rule code. 

This test could be considered the first point where the code is 
examined in a comprehensive, integrated manner. In general, more 
than one person is tasked with developing rule groups which 
creates situations for typical syntax inconsistencies to occur 
with respect an integrated rule base. Thus, this step in the 
testing process was considered to be particularly effective. 
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3.3.2 Functional Execution - Ordered Groups 
Analysis/Results : 

A decision not to perform this test was made for reasons similar 
to that discussed in section 3.2.4. The appropriateness of doing 
this test was outweighed by: 1) the partial satisfaction of test 
objectives by the test described in section 3.2.3. The sequencing 
mechanisms are, in part, implemented with control patterns within 
rule groups. Therefore, part of the original intent of this test 
had already been covered. In addition, it was found that 
effective results of salience type sequencing checks could only 
be obtained using realistic data, not canned "deffacts" default 
data. Further, such deffacts data would have required some 
"intrusive changes" to the rule sets to make the test situation 
function correctly (e.g., temporary modifications to rules to 
inhibit infinite loops situations) . 


3.3.3 Functional Execution - Unordered Groups 

Analysis/Results : 

This test was not performed for many of the same reasons as given 
above . 


3.4 INTERFACE TESTS 


3.4.1 Input File Sources 
Analysis/Results : 

This test was partially successful in meeting the test 
objectives. The repairs file used to assign a unique number to 
each of the data items sent to the expert system is provided in 
Appendix C. Examination of the results of running this test 
quickly showed an unanticipated problem with the specification of 
the test. The assignment of a unique number to each input into 
the expert system was done before the data preparation 
processing, not after as was implied in the test specification. 

The numeric form of the unique numbers was not consistent with 
the flag conversion, scaling, averaging, etc. performed in the 
data preparation process. Out of the data preparation process 
comes both symbolic values (e.g., high or low, which are no 
longer numbers and are difficult to correlated directly with the 
assigned unique number itself), and numeric values. 
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Therefore, the results of this test enabled the end-to-end data 
flow of some, not all, of the input data parameters to be 
examined. The results obtained were used as much as possible to 
trace the proper flow. The valid data passing through the data 
preparation process showed correct results. The checkoff of data 
sources, in support of the test described in section 3.4.2, was 
completed and successful. 


3.4.2 Default Data Sources 
Analysis/Results : 

The facts given in the listing from this test were used, along 
with the checkoff list from the test described in section 3.4.1, 
to further ensure that all data patterns in the rule base had 
some clearly defined source. Again, this test provided an 
opportunity to really understand what was in the rule base from 
the standpoint of code implementation. 


3.4.3 Non-HSTD Inputs 
Analysis/Results : 

This test was performed, but the procedure was done by code 
inspection rather than logging data in a dribble file as the 
original procedure specified. This was felt to be a more time 
efficient and effective method. The code which asserted each of 
the inputs was examined and the exact form of the fact that would 
be asserted was written down. This form was then compared to the 
pattern of the corresponding rule in the rule base. No 
discrepancies were found. 


3.4.4 HSTD Inputs 

Analysis/Results : 

A procedure similar to that described in section 3.4.3 was 
performed. 


3.4.5 Identify Function Calls 
Analysis/Results : 

This type of test is done inherently each time a rule set or sets 
is loaded into a CLIPS environment. Part of CLIPS' initial 
processing of rules is to make sure that all references to 
functions in the rules can be traced to the "defined functions" 
that CLIPS has access to. As a consequence of loading the rule 
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sets in previous tests, no unknown function references were 
indicated by CLIPS. Therefore, all function references were 
present as expected and this test, as originally defined, did not 
have to be performed. 


3.4.6 Function Call Returns 
Analysis/Results : 

This test was performed by inspection of the calling arguments in 
the rule function calls and the outputs of the defined C language 
functions defined for the expert system. This process was also 
verified as part of previous tests in that incorrect types of 
returned data from function calls will often result in a CLIPS 
syntax or data type or data usage error during execution. 


3.4.7 Status Light Indicators 
Analysis/Results : 

This test was performed by taking the "output. r" rule set and 
adding a test fact for each status light on the screen to the 
def facts section of the ruleset which was defining the light 
locations. This test was done for all status lights in each of 
three light modes: normal, inverse, and blink. Appendix D 
presents a listing of one of the three versions of the output 
rule set showing how the status light deffacts data was 
specified. 

Performing this test in this manner was a simplication over the 
original test definition because the status light indicators on 
the expert system screen all lit up essentially at the same time 
and stayed on the screen until the expert system was completely 
stopped. No anomalies were found. 


3.4.8 User Interface Command Acceptance 

Analysis/Results : 

The procedure for this test was altered and consisted of both 
inspection of the rules and user interface fact assertion code, 
and partially by the successful display of the appropriate 
configuration windows as illustrated in the ONAV Entry User's 
Guide (see applicable documents section 2.2). All inputs were 
found to be functioning properly. 
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3.5 ACCEPTANCE TESTS 


Due to the unavailability of experts required to evaluate such 
testing, which was outside the control of ONAV testing activities 
performed here, acceptance testing was not completed. Initial 
data files were made available with which to begin testing and 
previous testing activities resulted in the expert system being 
fully prepared for the detailed types of investigation and 
examination carried out during acceptance test activities. 

The test plan for acceptance testing remains fully valid and the 
appropriate organizations have been made cognizant of this 
situation. 
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Section 4 


CONCLUSIONS AND RECOMMENDATIONS 


4.1.1 Design Structure 

The design of the Entry expert system rule base, with numerous 
functional groups, was of significant benefit in both organizing 
and carrying out the tests that were performed. Such a breakdown 
of rule bases ensures that individual aspects of rules can be 
more easily tested, without having to deal with unnecessary 
interactions from or with other rules in the rule base. 


4.1.2 Development Techniques 

With respect to the accuracy of the coding effort, a data item 
dictionary (i.e., pattern name, format, and legal values) for the 
entire design would have been useful and would help avoid 
inconsistencies and inaccuracies in syntax and inter-rule-group 
integration. This type of item, while certainly an impact on 
testing of the rule base, is definitely a resource and time 
consideration during development activities. 


4.1.3 Testing Results 

The results of the testing performed demonstrated that the ONAV 
Entry expert system performs very well. Functional processing was 
found to be very complete and the various functional parts of the 
rule base were well integrated, both from a rule base standpoint 
as well as with non-rule components of the expert system (e.g., 
user interface) . 


4.1.4 Languages 

The use of CLIPS and C was very advantageous in the sense of 
using a consistent language throughout the expert system. 
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APPENDICES 


Batch file from test 4. 1.3.1 described in section 
3.2.3 of this document. 

Dribble file output from test 4. 1.3.1 described in 
section 3.2.3 of this document. 

Repair test file for test 4. 3. 1.1 described in 
section 3.4.1 of this document. 

Interface light rule set modified to support 
status light checks for test 4.3.2 described in 
section 3.4.7 of this document. 



APPENDIX 


Batch file from test 4. 1.3.1 described in section 
3.2.3 of this document. 
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APPENDIX 


Dribble file output from test 4. 1.3.1 described in 
section 3.2.3 of this document. 
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APPENDIX 


C - Repair 
section 


test file for test 4. 3. 1.1 described in 
3.4.1 of this document. 
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APPENDIX 


Interface light rule set modified to support 
status light checks for test 4.3.2 described in 
section 3.4.7 of this document. 
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